Extended abstract: Learning search strategies

The underlying motivation for the work presented in this paper is to usefully understand what is called the value of computation (Russell & Wefald 1991). By this intuitively mean the following. Suppose we have some computational process C which, at some point, has a choice between the computations co and cl. We would like to be able to make claims of the form: co is a better choice for C than cl because the value of co is greater than that of cl. Extending this idea by calling the set of choices made by C a program, we would similarly like to be able to say that the value of the program ~r = {ci } is greater than that of ~r’ = {e~}. This, in turn, would allow us to define the best, or bounded optimal (Russell & Subramanian 1995) program for C. Let us consider what would be required of a formalism to make these ideas meaningful. It is certainly necessary to first provide a context for the computation; i.e., we need to define what we want the computation to do. As an example, the classic framework of complexity has developed by considering the context of decision problems. Unfortunately, it is difficult to construct a definition for the value of computation within this framework which is ultimately non-trivial. The underlying cause of this difficulty is that there is a "correct answer" to a decision problem. This answer is independent of time, and is hence also independent of the computational process. At best, we can only discuss the amount of time (or some other resource) taken by the entire computational process which computes and outputs this answer. What we would like, instead, is to be able to consider the incremental process of computation and the tradeoffs between continuing and stopping the computation at any point in time. This idea is explored in the field of anytime algorithms (Dean & Boddy 1988; Zilberstein & Russell 1995). Unfortunately, a limitation of the framework of anytime algorithms is that the focus is still typically on some single, independent problem. It is assumed that this problem is given to the computational process when it is initialized, and the goal becomes that of characterizing the performance of the computation with respect to the time at which it is stopped/interrupted. What we would like, however, is to be able to express the problem itself as a process with which the computational process must interact; for example, we would like to be able to address the question of what to do when decision problems occur sequentially. Hence we find that the context of control problems is most suited for our discussion. The particular model of control problems which we consider is based on1 that of semi-Markov decision processes (SMDPs) (Parr 1998). It is well known SMDPs have optimal policies which are stationary 2.